Intelligently capturing digital images based on user preferences

ABSTRACT

Systems, methods, and computer program products to perform an operation comprising receiving image data provided by an image sensor, identifying, based on a facial recognition algorithm applied to the image data, a first face in the image data, identifying a plurality of rules applicable to capturing images based on the image data, and upon determining that a first rule of the plurality of rules restricts depiction of the first face in an image: performing a predefined operation to restrict depiction of the first face in the image data provided by the image sensor, and subsequent to performing the predefined operation to the image data, generating an image for output based on the image data, wherein the generated image does not depict the first face.

BACKGROUND

The present invention relates to digital image capture, and morespecifically, to intelligently capturing digital images based on userpreferences.

Cameras have become an integral part of daily life. With theproliferation of mobile devices that include cameras, most people carrya camera at all times. As such, users can capture images at any time.

SUMMARY

In one embodiment, a method comprises receiving image data provided byan image sensor, identifying, based on a facial recognition algorithmapplied to the image data, a first face in the image data, identifying aplurality of rules applicable to capturing images based on the imagedata, and upon determining that a first rule of the plurality of rulesrestricts depiction of the first face in an image: performing apredefined operation to restrict depiction of the first face in theimage data provided by the image sensor, and subsequent to performingthe predefined operation to the image data, generating an image foroutput based on the image data, wherein the generated image does notdepict the first face.

In another embodiment, a system comprises a processor and a memorystoring instructions, which when executed by the processor, performs anoperation comprising receiving image data provided by an image sensor,identifying, based on a facial recognition algorithm applied to theimage data, a first face in the image data, identifying a plurality ofrules applicable to capturing images based on the image data, and upondetermining that a first rule of the plurality of rules restrictsdepiction of the first face in an image: performing a predefinedoperation to restrict depiction of the first face in the image dataprovided by the image sensor, and subsequent to performing thepredefined operation to the image data, generating an image for outputbased on the image data, wherein the generated image does not depict thefirst face.

In another embodiment, a computer-readable storage medium hascomputer-readable program code embodied therewith, the computer-readableprogram code executable by a processor to perform an operationcomprising receiving image data provided by an image sensor,identifying, based on a facial recognition algorithm applied to theimage data, a first face in the image data, identifying a plurality ofrules applicable to capturing images based on the image data, and upondetermining that a first rule of the plurality of rules restrictsdepiction of the first face in an image: performing a predefinedoperation to restrict depiction of the first face in the image dataprovided by the image sensor, and subsequent to performing thepredefined operation to the image data, generating an image for outputbased on the image data, wherein the generated image does not depict thefirst face.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIGS. 1A-1B illustrate an example system architecture whichintelligently captures digital images based on user preferences,according to one embodiment.

FIG. 2 illustrates an example method to intelligently capture digitalimages based on user preferences, according to one embodiment.

FIG. 3 is a flow chart illustrating an example method to receive rulesfrom nearby devices, according to one embodiment.

FIG. 4 is a flow chart illustrating an example method to apply localrules to faces identified in image data prior to capturing an image,according to one embodiment.

FIG. 5 is a flow chart illustrating an example method to apply rulesreceived from nearby devices to faces identified in image data prior tocapturing an image, according to one embodiment.

FIG. 6 illustrates an example system which intelligently capturesdigital images based on user preferences, according to one embodiment.

DETAILED DESCRIPTION

Embodiments disclosed herein provide enhanced techniques for imagecapture that are based on the rules and/or settings of one or moreusers. For example, a user of a smartphone which includes a camera mayspecify in a user profile to only include the faces of people in animage for whom the user has stored personal contact information. Whenthe user attempts to capture an image using the smartphone, thesmartphone may perform facial recognition on all faces in the imagedata, and attempt to match each face in the image data to one of theuser's contacts (e.g., stored contacts, social media contacts, and thelike). If a face does not match any of the user's contacts, thesmartphone may perform a predefined operation to ensure that the face(and/or the body) is not depicted in a resultant image generated by thesmartphone user. For example, the predefined operation may includeapplying filters to blur the face, placing an object over the face, orany other type of operation to obfuscate the face, prior to capturingthe image.

In addition, embodiments disclosed herein may leverage rules of nearbyusers when capturing images. Generally, users may specify a set ofpersonal privacy rules and/or settings in their personal profile. Forexample, a first user may specify that they do not wish to be depictedin any image, regardless of who is capturing the image. A device with acamera may then receive the profile data (which may include an image ofthe respective user) from each nearby user via a wireless communicationinterface. When a user subsequently activates their camera, the devicemay perform facial recognition to determine whether each face in theimage data of the camera sensor is associated with a nearby user (basedon the user images received in the user profile). If a matching user isidentified, the rules associated with that user may be applied before animage is generated. For example, if the first user is identified, thefirst user's face may be removed or otherwise obscured in the image datasuch that a resultant image does not depict the first user's face.

FIG. 1A illustrates an example system architecture 100 whichintelligently captures digital images based on user preferences,according to one embodiment. As shown, the system 100 includes an imagecapture device 101. The image capture device 101 is representative ofany device which captures digital images, such as a digital camera,digital single-lens reflex (DSLR) camera, smartphone, tablet computer,laptop computer, portable gaming device, and the like. The system 100also includes three example computing devices 102-104. The computingdevices 102-104 are representative of any type of computing device whichcan wirelessly communicate with the image capture device 101, such as awearable device, digital camera, DSLR camera, smartphone, tabletcomputer, and the like.

As shown, a display 105 of the image capture device 101 outputs a visualdepiction of the image data captured by an image sensor of the imagecapture device 101. As shown, the display 105 depicts three people106-108 that would be depicted in an image generated by the imagecapture device 101 responsive to user input specifying to capture animage. However, a user of the image capture device 101 may not know oneor more of the people 106-108, and might not wish to have these peopledepicted in the images they capture (or generate). Furthermore, one ofthe people 106-108 may not like being photographed, and do not want tobe depicted in digital images generated by the image capture device 101.

Advantageously, the image capture device 101 is configured to applyuser-defined rules when capturing images such that the images depictonly those people that are desired to be included by the user capturingthe image, and only those people whose user preferences permit thedepiction of their likeness in the image. For example, a user of theimage capture device 101 may specify, in their user profile, a rule thatonly allows personal contacts to be depicted in the images generated bythe user. Therefore, when capturing images, the image capture device 101may process the image data provided by the image sensor of the imagecapture device 101 using a facial recognition algorithm. The facialrecognition algorithm may then match the faces of the people 106-108 toone or more data sources 125, such as a local cache of profile imagesstored on the image capture device, local image albums stored on theimage capture device 101, and remote data sources (such as social mediaplatforms, and the like). If the image capture device 101 identifies amatch between the face of the people 106-108, the image capture device101 may confirm that the person is a personal contact of the user of theimage capture device 101. For example the image capture device 101 maydetermine whether the match was based on a profile image of a contactstored in the image capture device 101, whether the match was based on aprofile image of a social media connection, and the like. However, ifthe image capture device 101 does not identify a match, or a matchingperson is not determined to be a personal contact of the user of theimage capture device 101, the image capture device 101 may perform apredefined operation to restrict the person 106-108 from being depictedin an image subsequently generated by the image capture device 101.

As shown, each device 102-104 stores a respective user profile 109-111.Each user profile may include a respective image 112-114 of theassociated user, and a set of rules 115-117 defined by each user. Asshown, for example, the rules 115 of user profile 109 specify that theuser has permitted the depiction of their image by permitting allimages. As another example, the rules 116 of user profile 110 specifythat no images of the associated user are permitted. As yet anotherexample, the rules 117 of user profile 111 specify that only personalcontacts are allowed to capture their image. Therefore, from theperspective of the image capture device 101, the rules and settingsdefined by a user of the image capture device 101 may be referred to“local rules,” and the rules and settings in the rules 115-117 may bereferred to as “remote rules.” Similarly, the rules 115 may be referredto as “local rules” from the perspective of the device 102, while therules 116-117 and rules associated with the image capture device 101 maybe referred to as “remote rules” from the perspective of the device 102.

When in proximity (e.g., when within a predefined distance), imagecapture device 101 may communicate with each of the devices 102-104.Generally, the devices 101-104 may use any method to determine whetherthe other devices 101-104 are within proximity. In at least oneembodiment, the image capture device 101 uses global positioning system(GPS) coordinates to determine whether the devices 102-104 are inproximity. In another embodiment, the image capture device 101 may usesignal strength of wireless signals to determine the respectivedistances from the devices 102. The image capture device 101 may thenreceive the profile data stored on each device 102-104, which includesthe images 112-114, the rules 115-117, and any other metadata stored inthe user profiles 109-111 (such as contact lists, social mediaconnections, preferences, and the like).

The image capture device 101 may use the received data from the devices102-104 when capturing images. Generally, the image capture device 101may compare the faces of the people 106-108 to the profile photos112-114 received from the devices 102-104. If the image capture device101 determines that a match exists, the image capture device 101 mayapply the rules 115-117 of the respective profiles 109-111 beforecapturing an image.

The boxes 118-120 depict example results of the comparison of the facesof the people 106-108 to the profile photos 112-114 performed by theimage capture device 101. As shown, the face of person 106 matches withthe profile photo 112 of profile 109, while the face of person 107matches the profile photo 114 of profile 111, and the face of person 108matches the profile photo 113 of profile 110. As such, the image capturedevice 101 must apply the local rules and the remote rules 115-117 tomodify the image data captured by the image sensor prior to generatingan image for output and/or storage.

FIG. 1B depicts an example image 121 generated by the image capturedevice 101 after applying local rules and the remote rules 115-117 priorto capturing the image 121. As shown on the display 105, the imagecapture device 101 has modified the image data provided by the imagesensor such that the person 106 is depicted, but persons 107-108 are notdepicted. For example, the image capture device 101 may camouflage thepersons 107-108, delete the image data corresponding to the location ofthe persons 107-108, distort the persons 107-108 using a filter appliedto the image data, and the like. Furthermore, the resultant image 121mirrors the display 105 in that person 106 is depicted in the image 121,while persons 107-108 are not depicted in the image 121.

For example, the image capture device 101 may determine, based on therules 115, that the person 106 can be depicted in images. However, thelocal rules for the image capture device 101 must also be satisfied.Therefore, the image capture device 101 may include the person 106 inthe generated image 121 because the person 106 is a personal contact ofthe user of the image capture device 101 and because the rules 115permit depiction of the person 106 in the image 121. Similarly, theimage capture device 101 may reference the rules 116 to determine thatperson 108 cannot be depicted in the image 121, as the rules 116explicitly state that the person 108 cannot be depicted in any images.Therefore, even if the person 108 was a personal contact of the user ofthe image capture device 101, the rules 116 would override the localrules in the image capture device 101, and the person 108 is notdepicted in the image 121. Furthermore, the image capture device 101 mayapply the rules 117 to determine that the person 107 permits their imageto be depicted only by known contacts. As shown in box 120, however, theuser of the image capture device 101 and the person 107 are notcontacts. Therefore, the image capture device 101 generates the image121 that does not include the person 107 based on the remote rules 117and the local rules of the image capture device 101.

FIG. 2 illustrates an example method 200 to intelligently capturedigital images based on user preferences, according to one embodiment.In at least one embodiment, the image capture device 101 performs thesteps of the method 200. As shown, the method 200 begins at block 210,where the image capture device 101 determines the local rules for a userof the image capture device 101. The rules may be stored in a userprofile, and may specify any rules related to image capture. Forexample, the user may specify that only family members should bedepicted in images taken in a first location, while second degreecontacts (e.g., friends of friends) and closer contacts should bedepicted in images taken at a second location. The user profile may bestored in the image capture device 101 or at a remote location (e.g.,the cloud). The rules may also specify what methods the image capturedevice 101 should use in obfuscating people from images (e.g., placingan object over the person, distorting their likeness, filling a portionof a bitmap for the image corresponding to the person with dummy data,and the like.

At block 220, described in greater detail with reference to FIG. 3, theimage capture device 101 may receive rules from nearby devices.Generally, at block 220, the image capture device 101 may wirelesslycommunicate with devices within a threshold distance to receive profiledata for participating users. The profile data may include an image ofeach user having a user profile on the nearby devices, as well as rulesand/or settings for each profile on the nearby devices. At block 230,the image capture device 101 may obtain image data from an image sensorof the image capture device 101. More generally, a user of the imagecapture device 101 may engage the camera feature of the image capturedevice 101, such as opening a camera application which displays theimage data captured by the image sensor, or turning on an image capturemode of a digital camera.

At block 240, the image capture device 101 may perform facialrecognition on the image data obtained from the image sensor to identifyfaces in the image data. The image capture device 101 may use any facialrecognition algorithm to analyze the image data and identify facestherein. At block 250, described in greater detail with reference toFIG. 4, the image capture device 101 may apply the rules of the user ofthe image capture device 101 to the faces identified in the image dataprior to capturing an image. Generally, at block 250, the image capturedevice 101 may determine whether to obfuscate each face identified atblock 240 based on the local rules. At block 260, described in greaterdetail with reference to FIG. 5, the image capture device 101 may applythe rules received from nearby devices to the faces identified in theimage data prior to capturing an image. Generally, at block 260, theimage capture device 101 may determine whether to obfuscate each faceidentified at block 240 based on the received rules. At block 270, theimage capture device 101 may capture an image based on the local andremote user rules applied to the image data.

FIG. 3 is a flow chart illustrating an example method 300 correspondingto block 220 to receive rules from nearby devices, according to oneembodiment. As shown, the method 300 begins at block 310, where theimage capture device 101 broadcasts a request to nearby devices. Theimage capture device 101 may use any wireless communications medium tocommunicate with the nearby devices, such as Bluetooth®, Wi-Fi®, nearfield communications (NFC), and the like. At block 320, the imagecapture device 101 may receive responses from one or more nearbydevices. The responses may specify profile data for one or more usersassociated with each nearby device. The profile data may include animage of each user, a plurality of image capture rules, and other usermetadata. In at least one embodiment, the image capture device 101 maydiscard responses from devices that are not within a threshold distanceof the image capture device 101.

At block 330, the image capture device 101 executes a loop includingblocks 340-370 for each device which responds to the image capturedevice 101 at block 320. At block 340, the image capture device 101receives the user profile data for one or more user profiles stored onthe current responding device. The profile data may include contactmetadata, metadata describing social media contacts, biographicalinformation of the user, preferred rules for sharing and/or storingimages depicting the user, and the like. At block 350, the image capturedevice 101 receives an image for each user having a user profile storedon the current responding device. At block 360, the image capture device101 may store the data received from the responding devices. At block370, the image capture device 101 determines whether more respondingdevices remain. If more responding devices remain, the image capturedevice 101 returns to block 330. Otherwise, the method 300 ends.

FIG. 4 is a flow chart illustrating an example method 400 correspondingto block 250 to apply rules for a local device to faces identified inimage data prior to capturing an image, according to one embodiment. Asshown, the method 400 begins at block 410, where the image capturedevice 101 executes a loop including blocks 420-470 for each faceidentified in the image data at block 240. In at least one embodiment,the image capture device 101 executes the loop including blocks 420-470in parallel for each face identified in the image data at block 240. Atblock 420, the image capture device 101 identifies the images ofcontacts and social media connections for the current user of the imagecapture device 101.

At block 430, the image capture device 101 may identify images stored onthe image capture device 101 and/or at a remote storage location (e.g.,the cloud). At block 440, the image capture device 101 determineswhether one or more images identified at blocks 420 and 430 matches thecurrent face identified in the image data. Generally, the match is basedon a comparison of the images to determine whether the person identifiedin each image is the same. If a match is not found, the image capturedevice 101 may proceed to block 470. However, in at least oneembodiment, the image capture device may proceed to block 445 even if amatch is not found, as the user of the current device may specify thatcapturing an image of unknown users is permitted.

Returning to block 440, if a match is found, the image capture device101 proceeds to block 445. At block 445, the image capture device 101determines whether the rules of the current user of the image capturedevice 101 permit depiction of the current face. Generally, at block445, the image capture device 101 determines whether the rules and rulesof the user of the image capture device 101 are satisfied by at leastone metadata (or profile) attribute of the profile of a matching user.If the rules permit depiction of the current face, the image capturedevice 101 proceeds to block 450. At block 450, the image capture device101 does not modify the image data corresponding to the current face.Generally, at block 450, the image capture device 101 does not modifythe image data because the current face in the image data corresponds toan image of a user that satisfies the rules specified by the user of theimage capture device 101. For example, the current face may belong tosomeone who is a contact or family member of the user of the imagecapture device 101, and the user of the image capture device 101 hasspecified a rule that only personal contacts or family members can bedepicted in the images generated by the image capture device 101.Similarly, the current face may belong to a social media contact who iswithin a specified degree of contacts specified by the user of the imagecapture device 101. However, as previously indicated, the rules maypermit depiction of the current face even if a match is not found atblock 440, as the profile of the user of the image capture device 101may permit depiction of faces that do not have a match.

Returning to block 445, if the rules specified in the user profile ofthe current user of the image capture device 101 do not permit depictionof the current face, the image capture device 101 proceeds to block 460.For example, metadata of the current person depicted in the image maynot satisfy the rules of the user of the image capture device 101. Forexample, the person depicted in the image may not be a contact of theuser of the image capture device 101, in violation of the rules of thecurrent user of the image capture device 101. At block 460, the imagecapture device 101 may perform a predefined operation to obfuscate thecurrent face. As previously indicated, the image capture device 101 mayperform any type of operation to obfuscate the current face, such asplacing an object over the current face, blurring the face, and thelike. At block 470, the image capture device 101 determines whether morefaces remain in the image data. If more faces remain, the image capturedevice 101 returns to block 410. If no more faces remain, the imagecapture device 101 proceeds to block 480, where the image capture device101 may cache the results of the comparisons performed at blocks 440 and445. Doing so allows the image capture device 101 to reference thestored data when subsequently determining whether to depict eachperson's face in an image based on the profile rules of the user of theimage capture device 101.

FIG. 5 is a flow chart illustrating an example method 500 correspondingto block 260 to apply received rules from nearby devices to facesidentified in image data prior to capturing an image, according to oneembodiment. As shown, the method 500 begins at block 510, where theimage capture device 101 executes a loop including blocks 520-580 foreach face identified in the image data at block 240. In at least oneembodiment, the image capture device 101 executes the loop includingblocks 520-580 in parallel for each face identified in the image data atblock 240. At block 520, the image capture device 101 executes a loopincluding blocks 530-570 for each user profile received at block 220,where each profile includes an image of an associated user associated.

At block 530, the image capture device 101 determines whether thecurrent face identified in the image data matches the image of the userassociated with the current user profile. If a match is found, the imagecapture device 101 proceeds to block 540. If a match is not found, theimage capture device 101 proceeds to block 570. At block 540, the imagecapture device 101 determines whether the rules specified in the userprofile including the image matching the current face permit capture ofthe user's image. If the rules permit capture of the user's image, theimage capture device 101 proceeds to block 550. At block 550, the imagecapture device 101 does not modify the image data corresponding to thecurrent face, as the identified user rules received from the remotedevices permit capture of an image of the user. The method may thenproceed to block 570.

Returning to block 540, if the rules received from the device of theuser matching the current face specify that an image of the user cannotbe depicted, the image capture device 101 proceeds to block 560. Forexample, the user may not want unknown persons to capture their image.As another example, the user may only want classmates to be able tocapture their image. At block 560, the image capture device 101 mayperform a predefined operation to obfuscate the current face. Aspreviously indicated, the image capture device 101 may perform any typeof operation to obfuscate the current face, such as placing an objectover the current face, blurring the face, and the like. The method maythen proceed to block 570.

At block 570, the image capture device 101 determines whether morenearby users remain. If more nearby users remain, the image capturedevice returns to block 520. If no more nearby users remain, the imagecapture device proceeds to block 580, where the image capture device 101determines whether more faces remain in the image data. If more facesremain, the image capture device 101 returns to block 510. If no morefaces remain, the image capture device 101 proceeds to block 590, wherethe image capture device 101 may cache the results of the comparisonsperformed at blocks 530 and 540. Doing so allows the image capturedevice 101 to reference the stored data when subsequently determiningwhether to depict each person's face in an image based on the profilerules of the user of the image capture device 101.

FIG. 6 illustrates an example system 600 which intelligently capturesdigital images based on user preferences, according to one embodiment.The networked system 600 includes a computer 602. The computer 602 mayalso be connected to other computers via a network 630. In general, thenetwork 630 may be a telecommunications network and/or a wide areanetwork (WAN). In a particular embodiment, the network 630 is theInternet.

The computer 602 generally includes a processor 604 which obtainsinstructions and data via a bus 620 from a memory 606 and/or a storage608. The computer 602 may also include a proximity module 617, one ormore network interface devices 618, camera 619, input devices 622, andoutput devices 624 connected to the bus 620. The computer 602 isgenerally under the control of an operating system (not shown). Examplesof operating systems include the UNIX operating system, versions of theMicrosoft Windows operating system, and distributions of the Linuxoperating system. (UNIX is a registered trademark of The Open Group inthe United States and other countries. Microsoft and Windows aretrademarks of Microsoft Corporation in the United States, othercountries, or both. Linux is a registered trademark of Linus Torvalds inthe United States, other countries, or both.) More generally, anyoperating system supporting the functions disclosed herein may be used.The processor 604 is a programmable logic device that performsinstruction, logic, and mathematical processing, and may berepresentative of one or more CPUs. The network interface device 618 maybe any type of network communications device allowing the computer 602to communicate with other computers via the network 630.

The storage 608 is representative of hard-disk drives, solid statedrives, flash memory devices, optical media and the like. Generally, thestorage 608 stores application programs and data for use by the computer602. In addition, the memory 606 and the storage 608 may be consideredto include memory physically located elsewhere; for example, on anothercomputer coupled to the computer 602 via the bus 620.

The proximity module 617 is representative of any type of module whichprovides location estimates for the computer 602. Examples of theproximity module 617 include GPS modules, Bluetooth radios, and thelike. The camera 619 is representative of any image capture device whichdigitally encodes image and videos based on data received from an imagesensor (not pictured). The input device 622 may be any device forproviding input to the computer 602. For example, a keyboard and/or amouse may be used. The input device 622 represents a wide variety ofinput devices, including keyboards, mice, controllers, and so on.Furthermore, the input device 622 may include a set of buttons, switchesor other physical device mechanisms for controlling the computer 602.The output device 624 may include output devices such as monitors, touchscreen displays, and so on.

As shown, the memory 606 contains the image application 612, whichfacilitates intelligent capture of images by the camera 619 based onuser preferences. Generally, the image application 612 may perform apredefined operation to obscure at least a portion of a person's body(e.g., the face, torso, etc.) upon determining one or more rules in theprofiles 615 do not permit depiction of the person in an image generatedby the camera 619. For example, a user of the computer 602, which may bea smartphone including a digital camera, may specify in the profiles 615to exclude the faces of unknown users in images generated by the camera619. The image application 612 may then identify faces in the image datacaptured by the image sensor of the camera 619, and perform facialrecognition on the faces to determine whether the faces are associatedwith known contacts (e.g., in an address book, social media connectionslists, and the like) of the user. If the faces do not belong to knowncontacts, the image application 612 may cause the camera 619 to performa predefined operation on image data provided by the image sensor priorto capturing the image. The predefined operation may be any type ofoperation, such as placing a user-specified object over the area wherethe unknown faces are present, scrambling the faces, or otherwiseobscuring the unknown faces.

Similarly, the image application 612 may receive and apply the rules ofusers of one or more nearby peer devices 650. For example, a user of afirst peer device 650 may specify to restrict their depiction in anygenerated image. The image application 612 may then receive the profile615 of the user of the first peer device 650, which may include an imageof the user. The image application 612 may then compare the image dataprovided by the image sensor of the camera 619 to the image of the userto determine whether the user is depicted in the image data. If the useris depicted in the image data, the image application 612 may cause thecamera 619 to perform a predefined operation to restrict depiction ofthe user of the first peer device 650 in an image subsequently generatedfor output by the camera 619.

As shown, the storage 608 contains the profiles 615 and the images 616.The profiles 615 stores profile data for one or more users. The profiledata stored in the profiles 615 may include an image of each user, rulesfor capturing an image of the associated user, and rules for who can bedepicted in images generated by the user. For example, a first user mayspecify that classmates and family members can capture their image, andthat images generated by the first user can include known contacts andfirst through third degree social media connections. The images 616stores images generated by the camera 619.

As shown, a plurality of peer devices 650 include an instance of theimage application 612 and a profiles 650. When the computer 602 iswithin a predefined distance (as determined based on the proximitymodule 617) of one or more of the peer devices 650, the computer 602 andpeer devices 650 may share data stored in the respective profiles 615.Doing so allows the image application 612 on the peer devices 650 andthe computer 602 to apply the rules of nearby users when capturingimages. Similarly, the peer devices 650 may share the profile datastored in the profiles 615 with each other.

Advantageously, embodiments disclosed herein provide techniques tocapture digital images based on user-specified rules stored in userprofiles. Users may specify who can be depicted in images they capture,as well as who can capture their image. Doing so enhances the field ofdigital imagery by respecting the personal privacy of users without theneed for constant manual intervention. Similarly, by performing apredefined operation to restrict depiction of a user prior to capturingan image, embodiments disclosed herein improve the functioning of imagecapture devices by reducing the storage space required to store images,as well as reducing the amount of post-processing required for generatedimages.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

In the foregoing, reference is made to embodiments presented in thisdisclosure. However, the scope of the present disclosure is not limitedto specific described embodiments. Instead, any combination of therecited features and elements, whether related to different embodimentsor not, is contemplated to implement and practice contemplatedembodiments. Furthermore, although embodiments disclosed herein mayachieve advantages over other possible solutions or over the prior art,whether or not a particular advantage is achieved by a given embodimentis not limiting of the scope of the present disclosure. Thus, therecited aspects, features, embodiments and advantages are merelyillustrative and are not considered elements or limitations of theappended claims except where explicitly recited in a claim(s). Likewise,reference to “the invention” shall not be construed as a generalizationof any inventive subject matter disclosed herein and shall not beconsidered to be an element or limitation of the appended claims exceptwhere explicitly recited in a claim(s).

Aspects of the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.”

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Embodiments of the invention may be provided to end users through acloud computing infrastructure. Cloud computing generally refers to theprovision of scalable computing resources as a service over a network.More formally, cloud computing may be defined as a computing capabilitythat provides an abstraction between the computing resource and itsunderlying technical architecture (e.g., servers, storage, networks),enabling convenient, on-demand network access to a shared pool ofconfigurable computing resources that can be rapidly provisioned andreleased with minimal management effort or service provider interaction.Thus, cloud computing allows a user to access virtual computingresources (e.g., storage, data, applications, and even completevirtualized computing systems) in “the cloud,” without regard for theunderlying physical systems (or locations of those systems) used toprovide the computing resources.

Typically, cloud computing resources are provided to a user on apay-per-use basis, where users are charged only for the computingresources actually used (e.g. an amount of storage space consumed by auser or a number of virtualized systems instantiated by the user). Auser can access any of the resources that reside in the cloud at anytime, and from anywhere across the Internet. In context of the presentinvention, a user may access applications or related data available inthe cloud. For example, the image application 612 could execute on acomputing system in the cloud. In such a case, the image application 612may store profile data for a plurality of users at a storage location inthe cloud. Doing so allows a user to access this information from anycomputing system attached to a network connected to the cloud (e.g., theInternet).

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

1. A method, comprising: receiving image data provided by an imagesensor of a device; identifying, based on a facial recognition algorithmapplied to the image data, a first face and a second face in the imagedata; identifying a plurality of image capture rules; determining that afirst rule of the plurality of image capture rules restricts depictionof the first face in an image; performing a predefined operation torestrict depiction of the first face in the image data provided by theimage sensor; determining that a second rule of the plurality of imagecapture rules permits depiction of the second face in images captured ata first predefined location, wherein the second rule further requires athreshold degree of relationship between a first user profile of thedevice and a second user profile associated with the second face;determining that the device is proximate to the first predefinedlocation; determining that the threshold degree of relationship existsbetween the first and second user profiles; and generating an image foroutput based on the image data, wherein the generated image depicts thesecond face and does not depict the first face.
 2. The method of claim1, further comprising prior to identifying the plurality of rules:determining that a plurality of user devices are within a predefineddistance of the device; receiving at least one user profile from each ofthe plurality of user devices, wherein each profile includes arespective user image associated with each profile; comparing the firstface in the image data to the plurality of user images; determining,based on the comparing, that the first face is associated with a thirduser profile of the received user profiles, wherein the first rule isspecified in the third user profile and specifies to restrict depictionof the first face in the generated image.
 3. The method of claim 2,further comprising: comparing the second face in the image data to theplurality of user images; determining, based on the comparing, that thesecond face is associated with the second user profile, of the pluralityof user profiles, wherein the second rule is specified in the seconduser profile; identifying the threshold degree of relationship betweenthe first and second user profiles; refraining from performing thepredefined operation to restrict depiction of the second face in theimage data provided by the image sensor; identifying, based on thefacial recognition algorithm applied to the image data, a third face inthe image data; comparing the third face in the image data to theplurality of user images; determining that a third rule of the pluralityof image capture rules requires the threshold degree of relationshipbetween a fourth user profile associated with the third face and thefirst user profile; determining that the threshold degree ofrelationship between the first and fourth user profiles does not exist;and performing the predefined operation to restrict depiction of thethird face in the image data provided by the image sensor, wherein thegenerated image does not depict the third face.
 4. The method of claim1, wherein the first rule is defined by a user of the device, the methodfurther comprising: comparing the first face in the image data to aplurality of user images, wherein each user image of the plurality ofuser images are received from one or more of: (i) the device, and (ii) aremote device, wherein determining that the first rule restrictsdepiction of the first face in the image comprises: determining that thefirst rule requires an association between a user profile of each persondepicted in the image data and the first user profile of the device;determining, based on the comparing, that the first face is associatedwith a third user profile, of a plurality of user profiles; anddetermining, that an association between the third user profile and thefirst user profile of the device does not exist.
 5. The method of claim4, further comprising: comparing the second face in the image data tothe plurality of user images; determining, based on the comparing, thatthe second face is associated with the second user profile, of theplurality of user profiles; identifying an association between thesecond user profile and the first user profile of the device; andrefraining from performing the predefined operation to restrictdepiction of the second face in the image data provided by the imagesensor.
 6. The method of claim 1, further comprising: prior toperforming the predefined operation, outputting the image data fordisplay on a display device of the device including, wherein the imagedata outputted on the display device depicts the first face; andsubsequent to performing the predefined operation, outputting the imagedata for display on the display device of the device, wherein the imagedata outputted on the display device does not depict the first facesubsequent to performing the predefined operation.
 7. The method ofclaim 1, wherein the predefined operation comprises one or more of: (i)scrambling the image data corresponding to the first face, (ii)overwriting the image data corresponding to the first face with imagedata corresponding to a predefined object, and (iii) overwriting theimage data corresponding to the first face with image data generated bya filter configured to blur image data.
 8. A computer program product,comprising: a non-transitory computer-readable storage medium havingcomputer readable program code embodied therewith, the computer readableprogram code executable by a processor to perform an operationcomprising: receiving image data provided by an image sensor of adevice; identifying, based on a facial recognition algorithm applied tothe image data, a first face and a second face in the image data;identifying a plurality of image capture rules; determining that a firstrule of the plurality of image capture rules restricts depiction of thefirst face in an image; performing a predefined operation to restrictdepiction of the first face in the image data provided by the imagesensor; determining that a second rule of the plurality of image capturerules permits depiction of the second face in images captured at a firstpredefined location, wherein the second rule further requires athreshold degree of relationship between a first user profile of thedevice and a second user profile associated with the second face;determining that the device is proximate to the first predefinedlocation; determining that the threshold degree of relationship existsbetween the first and second user profiles; and generating an image foroutput based on the image data, wherein the generated image depicts thesecond face and does not depict the first face.
 9. The computer programproduct of claim 8, the operation further comprising prior toidentifying the plurality of rules: determining that a plurality of userdevices are within a predefined distance of the device; receiving atleast one user profile from each of the plurality of user devices,wherein each profile includes a respective user image associated witheach profile; comparing the first face in the image data to theplurality of user images; determining, based on the comparing, that thefirst face is associated with a third user profile of the received userprofiles, wherein the first rule is specified in the third user profileand specifies to restrict depiction of the first face in the generatedimage.
 10. The computer program product of claim 9, the operationfurther comprising: comparing the second face in the image data to theplurality of user images; determining, based on the comparing, that thesecond face is associated with the second user profile, of the pluralityof user profiles, wherein the second rule is specified in the seconduser profile; identifying the threshold degree of relationship betweenthe first and second user profiles; refraining from performing thepredefined operation to restrict depiction of the second face in theimage data provided by the image sensor; identifying, based on thefacial recognition algorithm applied to the image data, a third face inthe image data; comparing the third face in the image data to theplurality of user images; determining that a third rule of the pluralityof image capture rules requires the threshold degree of relationshipbetween a fourth user profile associated with the third face and thefirst user profile; determining that the threshold degree ofrelationship between the first and fourth user profiles does not exist;and performing the predefined operation to restrict depiction of thethird face in the image data provided by the image sensor, wherein thegenerated image does not depict the third face.
 11. The computer programproduct of claim 8, wherein the first rule is defined by a user of thedevice, the operation further comprising: comparing the first face inthe image data to a plurality of user images, wherein each user image ofthe plurality of user images are received from one or more of: (i) thedevice, and (ii) a remote device, wherein determining that the firstrule restricts depiction of the first face in the image comprises:determining that the first rule requires an association between a userprofile of each person depicted in the image data and the first userprofile of the device; determining, based on the comparing, that thefirst face is associated with a third user profile, of a plurality ofuser profiles; and determining, that an association between the thirduser profile and the first user profile of the device does not exist.12. The computer program product of claim 11, the operation furthercomprising: comparing the second face in the image data to the pluralityof user images; determining, based on the comparing, that the secondface is associated with the second user profile, of the plurality ofuser profiles; identifying an association between the second userprofile and the first user profile of the device; and refraining fromperforming the predefined operation to restrict depiction of the secondface in the image data provided by the image sensor.
 13. The computerprogram product of claim 8, the operation further comprising: prior toperforming the predefined operation, outputting the image data fordisplay on a display device of the device, wherein the image dataoutputted on the display device depicts the first face; and subsequentto performing the predefined operation, outputting the image data fordisplay on the display device of the device, wherein the image dataoutputted on the display device does not depict the first facesubsequent to performing the predefined operation.
 14. The computerprogram product of claim 8, wherein the predefined operation comprisesone or more of: (i) scrambling the image data corresponding to the firstface, (ii) overwriting the image data corresponding to the first facewith image data corresponding to a predefined object, and (iii)overwriting the image data corresponding to the first face with imagedata generated by a filter configured to blur image data.
 15. A system,comprising: a processor; an image sensor; and a memory storing one ormore instructions which, when executed by the processor, performs anoperation comprising: receiving image data provided by an image sensorof a device; identifying, based on a facial recognition algorithmapplied to the image data, a first face and a second face in the imagedata; identifying a plurality of image capture rules; determining that afirst rule of the plurality of image capture rules restricts depictionof the first face in an image; performing a predefined operation torestrict depiction of the first face in the image data provided by theimage sensor; determining that a second rule of the plurality of imagecapture rules permits depiction of the second face in images captured ata first predefined location, wherein the second rule further requires athreshold degree of relationship between a first user profile of thedevice and a second user profile associated with the second face;determining that the device is proximate to the first predefinedlocation; determining that the threshold degree of relationship existsbetween the first and second user profiles; and generating an image foroutput based on the image data, wherein the generated image depicts thesecond face and does not depict the first face.
 16. The system of claim15, the operation further comprising prior to identifying the pluralityof rules: determining that a plurality of user devices are within apredefined distance of the device; receiving at least one user profilefrom each of the plurality of user devices, wherein each profileincludes a respective user image associated with each profile; comparingthe first face in the image data to the plurality of user images;determining, based on the comparing, that the first face is associatedwith a third user profile of the received user profiles, wherein thefirst rule is specified in the third user profile and specifies torestrict depiction of the first face in the generated image.
 17. Thesystem of claim 16, the operation further comprising: comparing thesecond face in the image data to the plurality of user images;determining, based on the comparing, that the second face is associatedwith the second user profile, of the plurality of user profiles, whereinthe second rule is specified in the second user profile; identifying thethreshold degree of relationship between the first and second userprofiles; refraining from performing the predefined operation torestrict depiction of the second face in the image data provided by theimage sensor; identifying, based on the facial recognition algorithmapplied to the image data, a third face in the image data; comparing thethird face in the image data to the plurality of user images;determining that a third rule of the plurality of image capture rulesrequires the threshold degree of relationship between a fourth userprofile associated with the third face and the first user profile;determining that the threshold degree of relationship between the firstand fourth user profiles does not exist; and performing the predefinedoperation to restrict depiction of the third face in the image dataprovided by the image sensor, wherein the generated image does notdepict the third face.
 18. The system of claim 15, wherein the firstrule is defined by a user of the system, the operation furthercomprising: comparing the first face in the image data to a plurality ofuser images, wherein each user image of the plurality of user images arereceived from one or more of: (i) the system, and (ii) a remote device,wherein determining that the first rule restricts depiction of the firstface in the image comprises: determining that the first rule requires anassociation between a user profile of each person depicted in the imagedata and the first user profile of the device; determining, based on thecomparing, that the first face is associated with a third user profile,of a plurality of user profiles; and determining, that an associationbetween the third user profile and the first user profile of the devicedoes not exist.
 19. The system of claim 15, the operation furthercomprising: comparing the second face in the image data to the pluralityof user images; determining, based on the comparing, that the secondface is associated with the second user profile, of the plurality ofuser profiles; identifying an association between the second userprofile and the first user profile of the device; and refraining fromperforming the predefined operation to restrict depiction of the secondface in the image data provided by the image sensor.
 20. The system ofclaim 15, wherein the predefined operation comprises one or more of: (i)scrambling the image data corresponding to the first face, (ii)overwriting the image data corresponding to the first face with imagedata corresponding to a predefined object, and (iii) overwriting theimage data corresponding to the first face with image data generated bya filter configured to blur image data, wherein the operation furthercomprises: prior to performing the predefined operation, outputting theimage data for display on a display device of the system, wherein theimage data outputted on the display device depicts the first face; andsubsequent to performing the predefined operation, outputting the imagedata for display on the display device of the device, wherein the imagedata outputted on the display device does not depict the first facesubsequent to performing the predefined operation.